eVoucher Impact Evaluation Endline Report

Reproducibility Package

Gabriel Caser dos Passos

Development Impact Evaluation (DIME)

Monday, the 11th of November, 2024

10/23/24 - Weekly Check in

  • Discussion with Matteo (10/21/24)
    • Replicated tables and figures may change the numbers a bit when comparing with Report and Presentation, since Matteo used different samples when producing some outputs;
    • It’s fine if the change is on the decimal level (Matteo expects that);
    • If results change a lot (more than decimals), I’ll flag Matteo to see what’s happening.
    • Folders seem to be well organized, but some results may be harder to replicate than others.
  • Next Steps
    • Currently I’m taking a look on the report to get familiar with it;
    • Replicate one figure or table and check with Matteo if the workflow is correct;
    • Start replicating all the other figures and tables for a week;
    • Give John an updated time frame of the whole task, considering MozSREP development.

10/30/24 - Weekly Check in

  • Changes in folder structure

    • Reorganized folders to store code and outputs following DIME Analytics standards

    • Created README file using DIME Analytics template (it’ll get updated along the way)

    • Working on the main dofile

  • Definition of tasks
    • Rewrite main scripts to run only necessary code (not the ones used during data collection, such as HFC)
    • Create folders with packages’ data, so replicators won’t need to install them on their computers (DIME A. standard)
  • Next steps
    • Identify, copy and successfully run all scripts required to produce the first figure
    • Define the workflow Send it to Matteo to get feedback

11/06/24 - Weekly Check in

  • Issues during replication

    • Figure 1

      • Different legend order
        • Matteo: Keep Overleaf order and change R code

    • Figures 2, 3 and 5

      • Couldn’t find code that creates figures files “ev_community_randomization.png”, ”ev_household_randomization.png” and”evch timeline3.png”

        • Matteo: They’re created on ppt. We don’t need to replicate them

11/06/24 - Weekly Check in (part 2)

  • Figure 4

    • getData() was removed from raster package, so we need to download the version used when the figure was created (not sure how) or change to another package, such as geodata and use function gadm()

      • Matteo: the second option, since finding the old version would be more complicated

        • I tried to use another package but couldn’t reproduce all the map details. Also tried to use the older version, but it doesn’t work at all. Me and Matteo agreed to schedule a meeting to solve it later.

      • Once we finish the replication package this kind of error won’t happen anymore since we now use renv (it stores all packages that used in a folder, preserving their versions for reproducers)

  • Done: Figures 1, 6, 7 and 8

  • Left: Figures 4, 9, 10 … 25

  • Not produced by code: Figures 2, 3 and 5

11/13/24 - Weekly Check in

  • Figure 13, 15 and 17 - Variables missing in the data

Figure 11 - Small differences in values (Left - Replicated ; Right - Overleaf)

  • Figure 21 - Small differences in values (Left - Replicated ; Right - Overleaf)

  • Table - Slightly different, but values are the same

11/13/24 Moz Team Meeting

  • Task: Reproducibility Package for eVoucher Endline Report’s Figures and Tables

  • Product: One script that reproduces all figures and tables in the report. For now, it considers data as given, so it only reproduces images, but not data itself.

Issues

  1. Visual differences, because of packages’ updates
  2. Data differences, because of ???
  3. Lack of data (for Maps), because of packages that download data from the internet and were discontinued

Solutions

  1. Changing code and using renv package (it stores all packages that were used in a folder, preserving their versions for reproducers)
  2. ???
  3. Changing code to use a different package and storing data locally.